feat(clean_slate): remove auto-simulated datasets by simulator provenance - #167
Conversation
…ance clean_slate.sh restored modified datasets but never reclaimed the simulated ones, so every workspace accumulated regenerable dataset/ payload indefinitely. Deleting "everything untracked under dataset/" would be wrong. Each workspace ignores dataset/ wholesale and force-adds real datasets back, so untracked does not imply regenerable: SMACS J0723 (52 MB) and a2744/data.fits are real data downloaded at runtime rather than redistributed -- the Mahler et al. Lenstool files are GPL-licensed and deliberately not committed. No intrinsic marker separates the two kinds either; README.md and tracer.json occur in both. So a dataset is removed only when a simulator script in the same repo demonstrably writes it: both its type and name appear as string literals in one scripts/**/simulator*.py or scripts/**/simulators/*.py. Everything else is kept, so the rule errs toward keeping (start_here-generated datasets survive). Scope is the 4 workspaces + 3 HowTo repos; autolens_profiling and autolens_jax_joss are excluded because their dataset/ dirs hold real instrument data (alma/sma/hst inputs, JWST cosmos_web_ring). Only directories are candidates -- loose untracked files inside a committed dataset (a2744/data.fits, double_einstein_ring/*.png) have generic names that would match almost any simulator. Also warns when a committed dataset file exceeds 5 MB, as a tripwire against large real data being added to a workspace. Verified: 88 paths removed across 7 repos (~13 MB); zero tracked files in the delete list; all 9 known-real datasets intact; all 7 repos report 0 dirty files afterwards, confirming only ignored paths were touched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012EZTtyLUAyKWuATyk4mkcm
…lators/ The simulator-script glob matched files named simulator*.py or any file under a directory named simulators/ (plural). HowToLens keeps its five simulators in scripts/simulator/ (singular), which matches neither — so the repo resolved to zero simulator scripts and the dataset sweep silently did nothing there. Surveyed all seven in-scope repos: HowToLens is the only one affected, and it is now covered (5 of its 6 dataset dirs removed; dataset/imaging/howtolens is written by the tutorial chapters rather than a simulator, so it is correctly kept). A missing spelling disables a whole repo without any error, so the glob now lists both directory forms explicitly and says why. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012EZTtyLUAyKWuATyk4mkcm
Correction: HowToLens does have simulator scriptsThe original description claimed HowToLens has none, so nothing there was cleaned. That was a bug in my glob, not a property of the repo. The pattern matched files named That matches neither form, so the repo resolved to zero simulator scripts and the sweep silently no-opped — no error, just an empty result. Fixed in 10d0708 by listing both directory spellings. I surveyed all seven in-scope repos; HowToLens is the only one affected (old pattern 0 scripts, new 6 — every other repo unchanged at 43/25/4/25/4/4). HowToLens now removes 5 of its 6 dataset dirs:
Re-verified after the fix: zero tracked files in the delete list, all 9 known-real datasets intact, HowToLens This is exactly the failure mode the rule is meant to have — a missing spelling disables an entire repo silently — so the glob now names both forms and carries a comment explaining why. |
…ipts "No simulator scripts found" and "this repo has nothing to clean" produced identical output — an empty result and a silent no-op. That is how HowToLens's scripts/simulator/ (singular) went unnoticed: the sweep reported nothing and looked correct. An in-scope repo with zero simulator scripts is almost certainly an unrecognised layout, so say so instead of no-opping quietly. Extracted the glob into simulator_scripts() so the sweep and the check share one definition and cannot drift apart. Verified by adding a simulator-less repo to DATASET_REPOS in a scratch copy: the warning fires. It stays silent for all seven real in-scope repos. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012EZTtyLUAyKWuATyk4mkcm
Added: empty-simulator-set warning (74e8749)Closing the silent-failure hole that let the HowToLens miss go unnoticed. The problem was that "no simulator scripts found" and "this repo has nothing to clean" produced identical output — an empty result and a silent no-op. The sweep reported nothing and looked correct. An in-scope repo with zero simulator scripts is almost certainly an unrecognised layout, so it now says so: Also extracted the glob into Verified both directions:
Safety checks still pass unchanged: zero tracked files in the delete list, all 9 known-real datasets protected, excluded repos absent. Branch state
|
1 participant
What
clean_slate.shrestored modified datasets but never reclaimed the simulated ones, so every workspace accumulated regenerabledataset/payload indefinitely. This adds a step 1b that removes them.Why not just "delete everything untracked under dataset/"
Because untracked does not mean regenerable. Every workspace ignores
dataset/**wholesale and force-adds the real datasets back via!dataset/<name>/**, so some real data is deliberately never committed:autolens_workspace/dataset/cluster/smacs0723/(52 MB)scripts/cluster/lenstool/data.py. The Mahler et al. Lenstool files are GPL-licensed and "downloaded at runtime with attribution rather than redistributed here" — committing them would be a redistribution problem.autolens_workspace/dataset/cluster/a2744/data.fitsautolens_workspace/dataset/cluster/csv_api_example/dataset/multi/rxj1131,dataset/weak/a2744_pyrrgI also tested two intrinsic markers hoping for a cheap discriminator — neither works.
README.mdis absent from real and simulated dirs alike, andtracer.jsonis present in real-input dirs (autolens_profiling/dataset/imaging/hst,.../interferometer/alma) while missing from genuinely simulated ones (simulated_lens,deblending).The rule
A dataset is removed only when a simulator script in the same repo demonstrably writes it: both its dataset type and its name appear as string literals in one
scripts/**/simulator*.pyorscripts/**/simulators/*.py. Anything without that provenance is kept, so the rule errs toward keeping —start_here.py-generated datasets survive uncleaned, which is the safe direction.Requiring both components was free: name-only and name+type matching gave identical results across all 7 repos, so the stricter form costs no coverage.
Two guards worth calling out:
dataset/cluster/a2744/data.fits— its generic name matches almost any simulator. Loose untracked files inside a committed dataset are never candidates.dataset/imaging/legitimately holds committed datasets (cosmos_web_ring) alongside simulated ones. It cut the delete list 89 → 62 and is deliberately not present.Scope
4 workspaces + 3 HowTo repos.
autolens_profilingandautolens_jax_jossare excluded outright — theirdataset/dirs hold real instrument data (alma/sma/hst inputs, JWSTcosmos_web_ring).HowToLenshas no simulator scripts, so nothing there is cleaned.Size tripwire
Warns when a committed dataset file exceeds 5 MB, guarding against large real data being added to a workspace. Currently fires on
autolens_workspace/dataset/cluster/simple/{data,noise_map}.fits(7 MB each, pre-existing).Verification
DRY_RUN=1previews unchanged;--packagingunchanged🤖 Generated with Claude Code
https://claude.ai/code/session_012EZTtyLUAyKWuATyk4mkcm